Skip to content

Asteroids! - #534

Open
mrosseel wants to merge 643 commits into
brickbots:mainfrom
mrosseel:feature/observable-asteroids
Open

mrosseel wants to merge 643 commits into
brickbots:mainfrom
mrosseel:feature/observable-asteroids

Conversation

@mrosseel

Copy link
Copy Markdown
Collaborator

Adds an observable asteroids dynamic catalog, mirroring the existing comet/planet catalogs.

  • Source: MPC Soft00Bright.txt (bright numbered asteroids), fetched via download_utils.
  • Propagation via Skyfield; filtered to currently visible objects, sortable by magnitude/distance.
  • Opposition-date info surfaced in object descriptions.
  • Menu + marking-menu entry, marker glyph, config migration, and test coverage.

Note on scope/diff size: this branch is stacked on deepchart → nixos → sqm, so the diff against main includes that underlying work (same as the deepchart/nixos PRs). The asteroid-specific change is ~35 files (asteroids.py, asteroid_catalog.py, catalog wiring, tests).

Opened to build a testable NixOS image and land the build in the unstable update channel.

🤖 Generated with Claude Code

https://claude.ai/code/session_01VaxurSrCzxrsfJrSjfA5bo

@mrosseel mrosseel added the testable Ready for testing via PiFinder software update label Jul 15, 2026
@mrosseel
mrosseel force-pushed the feature/observable-asteroids branch 4 times, most recently from f4f02d6 to b95cf39 Compare July 16, 2026 09:37
@mrosseel mrosseel changed the title feat(catalog): add observable asteroids Asteroids! Jul 16, 2026
Offline analysis of every referenced HQ sweep (2025-11-16 x2, 2025-11-18,
2026-07-16; full pipeline incl. per-sweep wing correction) shows a stable
-0.32 mag residual vs the hand-held reference meter (median of -0.28, -0.08,
-0.46, -0.36; spread 0.14). Raise the band offset 0.07 -> 0.39 to absorb it.

Method validated on imx462: the same offline pipeline reproduces on-device
SQM exactly and leaves imx462 residuals at +0.02 +/- 0.04 with its existing
+0.43 offset. Analysis artifacts: myPiFinder/support/dumps/analysis/20260716.
@mrosseel
mrosseel force-pushed the feature/observable-asteroids branch from 7b85bf2 to 3214d9a Compare July 16, 2026 12:57
…fsets

When a star's wings sit below the ring noise, the wing-boundary search cuts
at the first ring — at worst the aperture itself, where total==core makes the
enclosed fraction exactly 1.0 regardless of the optics. Faint fields then
fill the rolling window with these tautological 'perfect aperture' samples
and the wing correction collapses (measured: same-night HQ sweeps at f 0.92
vs 0.71, a 0.27 mag SQM split; 20% of the faint sweep's samples were exactly
degenerate).

Require the boundary to sit at least min_wing_radius out: a closer boundary
means the wings were not resolved for that star — a statement about the
star's brightness, not the optics — so the sample is dropped instead of
recorded as f~1. The threshold is one angular cut (~530 arcsec) expressed in
each sensor's photometry-image pixels (CameraProfile.wing_min_radius_px:
7 imx462/imx290 @490px green, 11 hq @760px, 15 imx296 @1088px mono), wired
in the solver when the SQM calculator is created.

Validated offline against all 14 archived sweeps (local cedar pipeline that
reproduces on-device SQM exactly): the same-night HQ pair converges from
f 0.92/0.71 to 0.64/0.67, and per-sweep f spreads tighten on both sensors.
The floor removes samples that previously diluted the correction, so wcorr
rises and both band offsets are recalibrated with it (they absorb the
estimator's mean and must move as a package):

- imx462/imx290: 0.43 -> 0.31 (6 referenced sweeps, residuals +/-0.10)
- hq:            0.39 -> 0.32 (3 independent references; the shared
  2025-11-16 reading sits +0.3 off on both its sweeps — suspect reference)

Analysis artifacts: myPiFinder/support/dumps/analysis/20260716.
@mrosseel
mrosseel force-pushed the feature/observable-asteroids branch from 3214d9a to f447639 Compare July 16, 2026 13:14
mrosseel added 4 commits July 16, 2026 15:14
- Cache the update manifest on disk and fetch refreshes in a background
  thread: the screen renders the last-known list immediately on entry,
  with a dim bottom-line indicator while checking (and 'update check
  failed' when offline with a cached list).
- Unstable rows lead with the bare PR number (fixed) followed by the PR
  title (scrolls when focused); store hashes no longer appear in the
  list. Trunk rows show the branch name after the dot.
- Focused rows and the confirm screen show build age ('built 5h ago')
  from a new built_at stamp CI now writes into every manifest entry.
- Trunk entries are only offered when their branch actually contains
  flake.nix (checked at refresh time, verdict cached in the manifest
  cache), so a non-NixOS upstream main never shows as installable.
- Cache the update manifest on disk and fetch refreshes in a background
  thread: the screen renders the last-known list immediately on entry,
  with a dim bottom-line indicator while checking (and 'update check
  failed' when offline with a cached list).
- Unstable rows lead with the bare PR number (fixed) followed by the PR
  title (scrolls when focused); store hashes no longer appear in the
  list. Trunk rows show the branch name after the dot.
- Focused rows and the confirm screen show build age ('built 5h ago')
  from a new built_at stamp CI now writes into every manifest entry.
- Trunk entries are only offered when their branch actually contains
  flake.nix (checked at refresh time, verdict cached in the manifest
  cache), so a non-NixOS upstream main never shows as installable.
@mrosseel
mrosseel force-pushed the feature/observable-asteroids branch from f447639 to c144fbd Compare July 16, 2026 15:44
@mrosseel
mrosseel force-pushed the feature/observable-asteroids branch from 4cbc7ec to 4200811 Compare July 16, 2026 16:14
@mrosseel
mrosseel force-pushed the feature/observable-asteroids branch from 4200811 to e665495 Compare July 16, 2026 16:45
@mrosseel
mrosseel force-pushed the feature/observable-asteroids branch from e665495 to de82f60 Compare July 16, 2026 16:46
mrosseel added 3 commits July 16, 2026 19:28
- Cache the update manifest on disk and fetch refreshes in a background
  thread: the screen renders the last-known list immediately on entry,
  with a dim bottom-line indicator while checking and on check failure.
  SQUARE re-fetches manually with the same feedback.
- Unstable rows lead with the bare PR number (fixed) followed by the PR
  title (scrolls when focused); the focused row and confirm screen show
  'built <age> · <hash>'. Trunk rows show the branch name after the dot
  and appear only when their branch carries flake.nix. Focused rows are
  bright, the rest dim.
- CI stamps built_at into every manifest entry (needs the same change on
  the branch pull_request_target reads trusted scripts from).
- Replace both 7x-square gestures with explicit menu items under
  Tools > Experimental > Dev Tools: a Dev Mode on/off toggle and a
  NixOS Migration entry (inert on systems already running NixOS).
- Rollback targets are validated like manifest entries so a misbehaving
  sys_utils cannot crash the screen.
@mrosseel
mrosseel force-pushed the feature/observable-asteroids branch from de82f60 to 190ac91 Compare July 16, 2026 17:28
mrosseel and others added 11 commits November 6, 2026 04:16
…rver (#117)

The brickbots DNS now points cache.pifinder.io and deltas.pifinder.io at
the same server as the .eu names. The devices, the CI and the docs use
the .io names from this change on.

The server keeps the .eu names as aliases. Devices that run an older
build still use them.

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
On rev 4, GPIO14 is the LTC2954 power-off kill line (ADR 0007). The
config.txt sets enable_uart=1, which makes GPIO14 the mini UART TXD. The
first byte that U-Boot prints cuts the power, so a migrated CM4 switches
itself off at each power-on.

- config.txt: a [cm4] section after [all] sets gpio=14=ip,pu. The
  firmware sets the pin back to an input with a pull-up before U-Boot
  starts. U-Boot needs enable_uart=1, and its serial drivers check only
  GPIO15, so it keeps its console. Brickbots tested this on a rev 4 unit:
  a cold boot stays on, reboot comes back, poweroff cuts the power.
- CM4 device tree: the gpio-poweroff overlay disables uart1. The kernel
  tree marked it okay with stdout-path on it, so the kernel registered
  ttyS0 on GPIO14 as its default console.
- SD images: nixos/sd-image.nix now holds the settings of nixpkgs'
  sd-image-aarch64.nix without its console=ttyS0 and console=ttyAMA0
  kernel parameters. A list option merges, so a later module cannot
  remove them. The image derivation diff shows only that change.

Already migrated rev 4 cards keep their old config.txt, because an
upgrade does not write the FAT partition.

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…igration guide (#119)

A release builds a migration tarball only when new_migration_tarball is
set. The tarball installs the newest release at first boot, so one tarball
serves many releases. Without a new tarball, update_manifest.py
--migration-from-manifest gives the entry the migration_v2 keys of the
newest release that has them. With no such release it stops with an error.

nixos/RELEASE.md now covers where each part is, the release rules, the
migration steps, the fixed contract with Pi OS 2.6.4, when a new tarball is
needed, and the known faults. ADR 0039 names migration_v2_url and the
shared tarball.

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* chore(nixos): kernel 6.18.60 (Raspberry Pi stable_20261103), update nixpkgs and nixos-hardware

The kernel tag and hash are fixed in nixos/pkgs/pifinder-kernel.nix, so
the kernel changes only when that file changes. nixos-hardware now
follows our nixpkgs.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix(nixos): btrfs checksum modules in the initrd for kernel 6.18

The forced initrd module list also removes the modules that the NixOS
btrfs module adds. Kernel 6.12 had crc32c built in. In 6.18 it is the
module crc32c-cryptoapi, so stage 1 could not mount the btrfs root.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* docs(nixos): kernel update steps and the initrd module fault

* docs(nixos): use git commands in the release guide

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…122)

nixos-hardware now sets sdImage.populateFirmwareCommands with mkForce.
Our value had the same priority, so both scripts ran. Its read-only
copies made our copy fail with Permission denied. mkOverride 40 keeps
only firmwareCommands.

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Developers, pre-commit and CI now run the same checks in the same flake
dev shell, built from python/uv.lock.

- python/scripts/check.sh runs each stage and enters the dev shell when
  needed. pytest runs with an empty temporary HOME, as on a CI runner.
- ruff 0.4.8 is in the uv dev group. uv uses the Nix environment and
  makes no .venv.
- nixos/libcamera-overlay.nix is used by the system and the dev shells,
  so both use the same libcamera.
- .github/actions/setup-nix is the one Nix and cache setup for all jobs.
- .github/scripts/push_dev_shell.sh pushes a dev shell's inputDerivation
  to the cache: nox (aarch64) and a new x86 job on green pushes, PR
  builds and releases.
- nox and the web tests run on ubuntu-24.04-arm. The web dev shell adds
  chromium and chromedriver.
- Tests: Blinka runs as a generic PC, every test gets sys_utils_fake,
  test_sys_utils.py tests the NixOS WiFi list and set_host_name, and the
  UI harness skips time.sleep in the main thread only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
python/uv.lock conflicted. It is the observable-asteroids lock, locked
again with uv lock: ruff 0.4.8 is new, no other version changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mrosseel and others added 13 commits November 15, 2026 04:40
Brings in 24 commits of brickbots main: the catalog import fixes
(brickbots#671 to brickbots#694), the Named Stars catalog (brickbots#695), the names view of the
object list (brickbots#696), the Pi OS side of the migration (brickbots#657), the release
workflow (brickbots#658), the OLED SPI clock (brickbots#701) and CI changes.

Conflicts and their resolution:
- Pi OS migration files (migration_gate.json, migration_progress,
  migration_progress.c, nixos_migration_calc.py): nixos keeps them
  deleted. They belong to the Pi OS line.
- noxfile.py, requirements_dev.txt: nixos keeps them deleted. The flake
  dev shell and uv2nix replace them.
- ui/software.py, tests/test_software.py, sys_utils.py, main.py: nixos
  keeps its own software screen. Its keep_awake and on_battery are the
  same feature as upstream's. The migration UI stays out of nixos.
- catalog_imports: upstream's code. The parser moved to designations.py,
  so the local copy of parse_designation goes. The Perek-Kohoutek loader
  stays and runs before the Named Stars loader.
- catalogs.py: upstream's NAME_MODE_CATALOGS. The T9 tables stay in
  catalog_arrays.py.
- object_details.py: upstream's listing index (brickbots#678), with a search on
  the listing_key column for an ObjectSequence.
- obslist.py and its tests: the name index of nixos (resolvers) with
  upstream's ambiguous designation check.
- tests/test_ui_modules.py: upstream's comet retry stub, the catalog
  build of nixos (no background loader).
- docs/ax/catalog.md, CONTEXT.md: text of both sides.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The three commits of brickbots release that main lacks: the back piece
with edge inserts, the BOM warning about the Adafruit 2472 BNO055 board
(brickbots#641) and the 2.6.2 pause of version.txt. The tree of main has the
same content already. version.txt stays 2.6.3.

Fixes for the merge of upstream main:
- object_details.py: the listing index in the format of ruff 0.4.8.
- star_identity.py: load_hipparcos copies the position columns. With
  copy-on-write, pandas gives a read-only array, and the fill of the
  unsolved positions failed.
- test_obslist_designations.py: the fake catalogs give iter_names, as
  the name index of nixos reads it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Brings the merge of brickbots main into nixos into deepchart.

Conflicts and their resolution:
- .github/workflows/nox.yml, web-integration-tests.yml: the aarch64
  runner and check.sh of deepchart. The ubuntu-24.04 pin of upstream
  is for setup-python, which these workflows do not use.
- ui/object_details.py: the image imports of deepchart, with
  listing_keys for the listing index of upstream (brickbots#678). One numpy
  import.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Brings the merge of brickbots main into deepchart into
observable-asteroids.

Conflicts and their resolution:
- ui/object_list.py: the imports of both sides (re for the names view
  of upstream, time for the asteroid branch).
- tests/test_ui_modules.py: the comet fixture keeps the
  check_if_comet_download_needed stub of the asteroid branch and adds
  the retry stub of upstream (brickbots#684).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…fixes

The committed database was built before the catalog import changes of
upstream main. It now holds the Named Stars catalog (578 stars) and the
import fixes (brickbots#681, brickbots#683, brickbots#685, brickbots#691, brickbots#692, brickbots#694, brickbots#695). EGC has 102
objects (was 91) and SaM 2157 (was 2162). IC 1805 is now one object with
Sh2 190 and Lyn 77, of type cluster with nebula. The catalog data test
has the new values.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Harris and Lynga loaders parsed their float columns as float32, and
arp.sqlite stores its magnitudes as float32 values, so the database held
values such as 6.46999979019165 for M 2. The Harris and Lynga loaders now
parse float64, and the Arp loader rounds to two decimals. The database is
rebuilt.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

testable Ready for testing via PiFinder software update

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants